Systems And Methods For Software-Based Video Conference Camera Lighting

ABSTRACT

A computer-implemented lighting system for video recording and display includes: a first user device featuring a front-facing camera, a front-facing display, a processor, a memory, and a networking interface; and a second user device featuring a front-facing camera, a front-facing display, a processor, a memory, and a networking interface; wherein the first user device&#39;s processor: receives a first video captured by the second user device&#39;s front-facing camera; displays the first video on the first user device&#39;s display, wherein a portion of the first video is replaced by one or more lighting elements; and captures a second video using the first user device&#39;s front-facing camera, wherein the second video is illuminated during capture, in part, by the one or more lighting elements.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. Application No.63/244,569 filed Sep. 15, 2021, which is hereby incorporated byreference.

BACKGROUND OF THE INVENTION

The present subject matter relates generally to systems and methods forproviding a software-based video conference camera lighting. Morespecifically, the present invention relates to systems and methods in avideo conference between a first participant and a second participantfor using portions of the first participant's display to providelighting of the first participant.

More than a billion smartphone users worldwide have a front-facingselfie camera on their mobile device. The “selfie” is a key part ofsmartphone photography, video capture, and video conferencing.

Front-facing cameras (i.e., cameras capable of capturing an image of auser that is looking at the display of the device) often do not havetheir own separate flash/lighting modules and, therefore, do not havethe hardware required to produce high quality photographs or videos inlow-level lighting or other difficult lighting conditions. To addressthis hardware limitation, there are a number of current systems andmethods for selfie photography that turn the entire smartphone displayscreen white as a photograph is taken to cast light onto thescene/person whose image is being captured.

The drawback of the “flashing white screen” solution is that the usercannot see the displayed image at the moment of capture (arguably one ofthe primary benefits of using a selfie/front facing camera), which is aneven more significant issue when using the front-facing camera tocapture video, as the display would need to be in a continuous whitescreen state preventing the user from using the display to align thescene they wish to capture. This solution also cannot be used in videoconferencing without entirely eliminating the first participant'sability to see the second participant.

Accordingly, there is a need for systems and methods for improvinglighting conditions of photographs and videos taken using a front-facingcamera by providing software-based video conference camera lighting inwhich a first participant's video display shows a live video feed of asecond participant, while additionally using at least a portion of thefirst participant's video display to light the first participant, asdescribed herein.

BRIEF SUMMARY OF THE INVENTION

To meet the needs described above and others, the present disclosureprovides systems and methods for software-based solutions for improvinglighting conditions of photographs and videos taken using a front-facingcamera, i.e., the lighting system. In a primary embodiment of thelighting system, a portion of a mobile device display is used to show animage of the scene being captured by a front-facing camera and/or a livevideo feed of another participant in a video conference while anotherportion of the mobile device display is used to light the scene beingcaptured. Although the primary embodiment of the lighting systemdescribed herein relates to the operation of a mobile device (e.g.,smartphone, tablet, dedicated video mobile calling devices such as theone sold by Facebook under the trademark Portal, etc.), it is understoodthat the teachings provided are equally applicable to any situation inwhich a user is capturing an image or video of themselves while viewinga display, including personal computer and laptop systems, office-basedteleconferencing systems, etc., as will be understood by those skilledin the art based on the descriptions provided herein.

A first example illustrating an application of the lighting systemtaught herein includes two users participating in a video conferencewith each other using their smartphones. In this example, each usercommunicates with the other using a smartphone running video conferencesoftware enabling each user to view a live video feed of the other useron a front facing display of their device while simultaneously capturinga live video feed of themselves using a front facing camera of theirdevice. As used herein, the term “front facing” refers to the locationof an element (e.g., a display, a camera, etc.) located on the displayside of a device such that the element faces the user when the user isfacing the display. For example, a typical smartphone or tablet includesa front facing display (e.g., touchscreen display), a front facingcamera (e.g., selfie camera), and a rear facing camera (e.g., primarycamera). A typical laptop computer includes a front facing display(e.g., laptop display screen) and a front facing camera (e.g.,integrated web camera). A typical desktop personal computer that isconfigured for video conferencing includes a front facing display (e.g.,monitor) and a front facing camera (e.g., web camera).

In this first example, each user's chat software enables their device touse the display lighting functionality taught herein. For example, ifthe lighting conditions for the first user are such that the video feedof the first user captured by the first device are sub-optimal (e.g.,the first user is in a low light or dark environment), the first usermay employ display lighting to improve the lighting conditions for thecapture of the video feed. To do so, the system identifies areas of thefirst display that are to be used for lighting while displaying the livevideo feed of the second user captured by the second device.

In one embodiment, the lighting system may identify the areas of thefirst display to be used for lighting based on a user selection. In someinstances, the first user (who is viewing the display that is to be usedfor lighting) makes a selection as to the areas of the first displaythat are to be used for lighting. In another instances, the second user(whose video is being viewed by the first user) makes a selection as tothe areas of the first display that are to be used for lighting. Inother instances, the selection of the areas of the first display thatare to be used for lighting analysis may be completed by the system,whether by a processor of the first user's device, a processor of thesecond user's device, or a processor in another system device, such as aserver through which the videos are routed between the two user devices.

Segmentation of the video feed (i.e., the separation of an object from abackground in a video) may enable more precise and more naturallyintegrated feeling lighting schemes, as will be described herein. Anysuitable segmentation method may be used to accomplish the advantagesand objectives of the subject matter presented herein, including, forexample, using computer vision, semantic segmentation, Al-basedsegmentation, instance segmentation, K-clustering, edge detection,kernel density estimation, Gaussian mixture model, etc.).

In some examples, the areas of the first display that are to be used forlighting may be bars (i.e., strips) located along one or more edges ofthe display. In other examples, the areas of the first display that areto be used for lighting may be circular “bulb-like” areas forming aperimeter around one or more edges of the display. Other geometricshapes may be used for the lighting, as will be understood by thoseskilled in the art based on the teachings provided herein.

In some examples, the lighting may be overlaid on the foreground of thevideo feed to maximize the lighting effect. In other examples, thelighting bars may be layered into the background of the video feedbehind a segmented first layer (e.g., a person shown as the subject inthe live video feed) so as to not obscure, and to give priority to, theprincipal content of the video feed.

In other examples, the areas of the first display that are to be usedfor lighting may be the portions of the display that would otherwise bedisplaying an identified background segment of the video. For example,in a typical selfie-style video chat between two users, the image ofeach user may be segmented from the background and the entirety of thesegmented background may then be replaced by lighting.

The lighting system may enable users to turn on or off the displaylighting (i.e., lighting filter) manually. In some instances, such aswhen the lighting system recognizes there is poor lighting by analyzingthe lighting conditions being captured by either front facing or rearfacing cameras, users may be prompted to engage the lighting filter. Inother instances, a lighting filter may automatically be applied by thesystem in response to ambient lighting environments identified by theone or more cameras.

To further illustrate the lighting system provided herein, additionalexamples of the system and its uses are described. In one arrangement, afront facing camera of a first user's mobile device (e.g., smartphone)is activated and begins capturing video as part of a telecommunicationconnection between two or more users. The captured video is displayed ona second user's mobile device display (e.g., smartphone touchscreendisplay) and in a picture-in-picture (PIP) window shown on the firstuser's mobile device display. In order to improve the lightingconditions for the video being captured of the first user by the firstuser's mobile device, lighting effects including a border of brightand/or white light shines around the edge of the display screen of thefirst user's mobile device illuminating the first caller. Depending onthe arrangement of the lighting system, the second caller may or may notbe able to see the white light border on the display of the second userdevice. For example, the PIP window of the second user's mobile devicemay show the live video feed of the second user as well as the lightingeffects treatment applied by the first user device. This arrangementenables the second user to more clearly visualize what is beingdisplayed to the first user (e.g., the video and the lighting effects).

In some examples, both of the participants in the video call may be inlow-light or dark environments and each may activate lighting effects ontheir mobile devices simultaneously to improve their respectivelighting. In such an example, each user's device may include one or morelighting effects and each user's device may show a PIP window that showsthe live video feed being sent to the other user, as well as thelighting effects treatment applied by the user's device so that eachuser can more clearly visualize what is being displayed to each user.

In other embodiments, the location and shape the lighting effects may besomething other than a simple border. The illumination may be providedusing any shapes, including, for example, one or more dedicated lightingsections at the top or bottom of the screen. In another example, aring-type shape may be provided in the middle of the display screen.

It is contemplated that the lighting shapes/configurations may be userselected options or may be determined by the lighting system based on ananalysis of the lighting conditions and/or the content of the videobeing captured. For example, the lighting system may be able to identifythe magnitude of lighting effects required to optimize the video capturebased on ambient lighting information acquired through the one or morecameras and/or analysis of the video being captured, for example, byevaluating the contrast level, shadows, etc. in the captured video. Whenthe lighting demands are determined to be greater (i.e., lower qualityambient light, lower quantity ambient light, etc.), the lighting systemmay use a greater proportion of the display for lighting than it wouldwhen the lighting demands are determined to be lesser.

In another embodiment, the lighting shapes/configurations are determinedby segmenting the object(s) of interest from the background in thecaptured video. In some examples, this segmentation is enabled by theuse of a spatial, distance, or depth-sensing camera (i.e., range camera,time-of-flight camera, LIDAR, the camera sold by Apple Inc. under theregistered trademark TrueDepth, etc.) or similar as the front facingcamera. Generally speaking, the object of interest is primarily going tobe the first user. However, the object of interest may be another object(animate or inanimate), such as a pet or a piano. When segmented, thebackground shape identified behind the object of interest can be used toprovide the lighting. For example, the entirety of the segmentedbackground may be replaced by a bright/and or white light to illuminatethe video image of the other user viewing the display. In someembodiments, both users' captured videos are segmented into object ofinterest and background segments and both of the background segments arereplaced by lighting effects. In other embodiments, only one of theusers' captured videos are segmented into object of interest andbackground segments and replaced by lighting effects.

It is contemplated that the PIP display for each device may or may notshow the lighting effects being applied on the other device. Displayingthe lighting effects employed by the other device enables the PIPviewing user to see the exact image that is being displayed to the otheruser(s).

It is further contemplated that the lighting effects may or may notoverlap the PIP window image. In other words, in some instances, thelighting effects will be further in the foreground than the PIP windowand, in other instances, the lighting effects will be further in thebackground than the PIP window. A user setting may enable prioritizationof the PIP window or the lighting effects.

In addition to providing various options as to the size, the shape, andthe location of the lighting effects, the brightness of the lightingeffects may be varied. Similarly, the color, or temperature, of thelighting effects may be varied. The brightness and color of the lightingeffects may be adjusted manually or automatically by the system based onan analysis of the ambient conditions and/or the quality of the capturedvideo. The brightness and temperature of the lighting effects may beindependently controlled relative to other displayed elements such thatthe system may, for example, increase or decrease the brightness of thelighting effects without affecting the brightness of the video feed.

In addition to the examples provided above describing how the lightingeffects are provided by manipulating various portions of the displayscreen, it is contemplated that one or more lighting modules may beprovided in addition to the user device display to substitute for or tosupplement the lighting effects described above. For example an LEDlight ring may be provided around a display along the border of asmartphone. Such additional lighting modules may be integrated into thesystem such that the lighting system is able to turn on and offadditional lighting modules as needed to replace or complement thelighting effects on the display. Control of the additional lightingmodules may be manual or automated by the lighting system. Similar tothe control of the lighting effects, the control of the additionallighting may be automated based on an analysis of the ambient lightingconditions and/or an analysis of the captured video. The integration ofadditional lighting modules provides a greater range of options for thesystem to illuminate the users. The lighting system may be manuallycontrolled by the user or may be automatically controlled by the systemto: (i) provide no lighting; (ii) provide lighting using only the devicedisplay; (iii) provide lighting using only the lighting modules; or (iv)provide lighting using both the device display and the lighting modules.

While primarily described above with respect to video conference calls,the solutions provided herein can be adapted and applied to selfiephotography and videography (i.e., a user taking a picture or video ofthemselves using a front facing camera). In these instances, thelighting effects are used to improve the lighting of a photograph orvideo, typically to be captured and stored by the user device, but areotherwise nearly perfectly analogous to the examples provided above.

For example, foreground lighting elements may be used to providelighting from the device's display to illuminate the user when using thedevice's front facing camera for selfie photography and videography.Similarly, the lighting elements may be background lighting elements(i.e., may not overlap the image of the user on the display) or may evenoccupy an entirety of a segmented background area of the display whenusing the device's front facing camera for selfie photography andvideography.

When saving a photograph or video to the device's memory, the capturedcontent may be stored such that it includes a representation of thelighting effects that were used at the time of capture (e.g., thelighting effects are superimposed on the photograph or video), such thatthe captured content does not include a representation of the lightingeffects, or such that the device stores a copy of each version of thecaptured content. For example, when reviewing a stored photograph, auser may be able to toggle between two versions of the photograph, oneincluding the lighting effects and one without. In some embodiments, thesaved image may provide the user an option to turn on and off thelighting effect while viewing a saved photograph.

In some examples, the display may provide a picture-in-picture displaythat may or may not show the lighting effects. For example, the mainportion of the display may show the user and the lighting effects whilethe picture-in-picture shows the content that is to be captured andstored without the lighting effects. In another example, the mainportion of the display may show the user and the lighting effects whilethe picture-in-picture shows the content that would have been takenbefore the lighting effects were applied. Like the lighting effectsthemselves, these picture-in-picture representations may be captured andstored as part of the photograph or video file such that the user maylater view the photograph or video with or without thepicture-in-picture display.

Another feature that may be more relevant for selfie photography andvideography, rather than for video conference calls (though could beused in connection with either), is the ability to manually orautomatically turn off and on the lighting effect while preparing forand then capturing an image or video. For example, the user device,through the front facing camera, may frame a user and a background.Within the framed image, the user device may segment the image into anobject of interest (i.e., the user) and a background. The lightingeffects may be configured to replace the background with a bright whiteimage. However, when lining the photograph up, the actual background isdisplayed to the user (i.e., without the lighting effects) and then, inresponse to the user taking action to turn on the lighting effects ortaking action to capture the photograph or video, the segmentedbackground is replaced with the lighting effects such that theyilluminate the user. In this manner, the user is able to properly framethe photograph or video to be captured, including the background, andthe lighting effects are able to illuminate the user to improve thequality of the lighting of the content captured.

The lighting system may provide visual feedback to a user through thedevice's display, such as an image, an icon, or text, to communicate tothe user whether or not any lighting effects will be applied to thecontent to be captured. Similarly an image, icon, or text may bedisplayed in connection with stored content to communicate whether thecontent was stored using one or more lighting effects.

As described herein, in some embodiments, the lighting system willautomatically suggest or automatically apply one or more lightingeffects in response to the ambient environment. For example, when thedevice determines that the quality of the lighting in the content to becaptured would be improved by providing one or more lighting effects(for example, by observing the ambient lighting conditions using thefront facing camera), the lighting system may present the user with anoption to activate one or more lighting elements or to authorize thelighting elements to be automatically activated by the lighting system.

In one example, a computer-implemented lighting system for videorecording and display includes: a first user device featuring afront-facing camera, a front-facing display, a processor, a memory, anda networking interface; and a second user device featuring afront-facing camera, a front-facing display, a processor, a memory, anda networking interface; wherein the first user device's processor:receives a first video captured by the second user device's front-facingcamera; displays the first video on the first user device's display,wherein a portion of the first video is replaced by one or more lightingelements on the first user device's display; and captures a second videousing the first user device's front-facing camera, wherein the secondvideo is illuminated during capture, in part, by the one or morelighting elements on the first user device's display.

The first video may include video of an object of interest (e.g., thesecond user) and a background and one of the first user device'sprocessor and the second user device's processor: identifies andsegments the object of interest from the background in the first video;and replaces at least a portion of the segmented background in the firstvideo with the one or more lighting elements.

The operation of the one or more lighting elements on the first userdevice's display may be user controlled or may be automaticallycontrolled by the first user device's processor.

In some examples, the second user device's processor: receives a secondvideo captured by the first user device's front-facing camera; displaysthe second video on the second user device's display, wherein a portionof the second video is replaced by a second set of one or more lightingelements on the second user device's display; and captures a first videousing the second user device's front-facing camera, wherein the firstvideo is illuminated during capture, in part, by the second set of oneor more lighting elements on the second user device's display.

The lighting elements may overlap a picture-in-picture element shown onthe first user device's display.

The lighting elements displayed on the first user device's display maybe replicated on a picture-in-picture display on the second userdevice's display such that the picture-in-picture display on the seconduser device's display replicates the first user device's display.

The lighting elements displayed on the first user device's display maynot be replicated on a picture-in-picture display on the second userdevice's display such that the picture-in-picture display on the seconduser device's display shows the first video without any representationof the lighting elements displayed on the first user device's display.

A picture-in-picture display on the second user device's display mayindicate the operation of the one or more lighting elements on the firstuser device's display.

A brightness of the one or more lighting elements on the first userdevice's display automatically adjust based on ambient lightingconditions detected by the first user device.

A brightness of the one or more lighting elements on the first userdevice's display and a brightness of the first video displayed on thefirst user device's display may be independently adjustable.

A brightness of the second user device's may be controlled by the firstuser device's processor.

The one or more lighting elements on the first user device's display mayreplace a segmented background element in the first video.

The one or more lighting elements on the first user device's display maybe displayed behind a segmented head and shoulders captured by thesecond user device's front-facing camera.

In another example, a computer-implemented lighting system for videorecording and display includes: a first user device featuring afront-facing camera, a front-facing display, a processor, a memory, anda networking interface; and a second user device featuring afront-facing camera, a front-facing display, a processor, a memory, anda networking interface; wherein the first user device's processor:receives a first video captured by the second user device's front-facingcamera; displays the first video on the first user device's display,wherein a portion of the first video is replaced by a first set of oneor more lighting elements on the first user device's display;automatically adjusts a brightness of the first set of one or morelighting elements on the first user device's display based on ambientlighting conditions detected by the first user device; and captures asecond video using the first user device's front-facing camera, whereinthe second video is illuminated during capture, in part, by the one ormore lighting elements on the first user device's display; wherein thesecond user device's processor: receives a second video captured by thefirst user device's front-facing camera; displays the second video onthe second user device's display, wherein a portion of the second videois replaced by a second set of one or more lighting elements on thesecond user device's display; automatically adjusts a brightness of thesecond set of one or more lighting elements on the second user device'sdisplay based on ambient lighting conditions detected by the second userdevice; and captures a first video using the second user device'sfront-facing camera, wherein the first video is illuminated duringcapture, in part, by the one or more lighting elements on the seconduser device's display.

In some embodiments, the first video includes video of an object ofinterest and a background and one of the first user device's processorand the second user device's processor: identifies and segments theobject of interest from the background in the first video; and replacesat least a portion of the segmented background in the first video withthe first set of one or more lighting elements; and the second videoincludes video of an object of interest and a background and one of thefirst user device's processor and the second user device's processor:identifies and segments the object of interest from the background inthe second video; and replaces at least a portion of the segmentedbackground in the second video with the second set of one or morelighting elements.

A brightness of the first set of one or more lighting elements on thefirst user device's display and a brightness of the first videodisplayed on the first user device's display may be independentlyadjustable and a brightness of the second set of one or more lightingelements on the second user device's display and a brightness of thesecond video displayed on the second user device's display may beindependently adjustable.

A brightness of the first set of one or more lighting elements may beadditionally controlled by the second user device's processor and abrightness of the second set of one or more lighting elements may beadditionally controlled by the first user device's processor.

A brightness of the first set of one or more lighting elements may beadditionally controlled by the first user device's processor and abrightness of the second set of one or more lighting elements may beadditionally controlled by the second user device's processor.

The first set of one or more lighting elements on the first userdevice's display may be displayed behind a segmented head and shoulderscaptured by the second user device's front-facing camera and the secondset of one or more lighting elements on the second user device's displaymay be displayed behind a segmented head and shoulders captured by thefirst user device's front-facing camera.

The first set of one or more lighting elements displayed on the firstuser device's display may be replicated on a picture-in-picture displayon the second user device's display such that the picture-in-picturedisplay on the second user device's display replicates the first userdevice's display and second set of one or more lighting elementsdisplayed on the second user device's display may be replicated on apicture-in-picture display on the first user device's display such thatthe picture-in-picture display on the first user device's displayreplicates the second user device's display.

An object of the invention is to provide an improved lighting system forcapturing video using the front facing camera of a device while lightingthe scene using portions of the associated display.

Another object of the invention is to improve the quality of videoimages captured in low-light and dark environments.

Another object of the invention is to provide a convenient solution forremoving shadows when communicating using the video chat function of amobile device.

Another object of the invention is to replace a portion of a displayedvideo on a front-facing display with lighting elements that provide abeatification filer when capturing video with a front facing camera.

Additional objects, advantages, and novel features of the solutionsprovided herein will be recognized by those skilled in the art based onthe following detail description and claims, as well as the accompanyingdrawings, and/or may be learned by production or operation of theexamples provided herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawing figures depict one or more implementations in accord withthe present concepts, by way of example only, not by way of limitations.In the figures, like reference numerals refer to the same or similarelements.

FIG. 1 is a schematic drawing of elements of a computer-implementedlighting system for video recording and display.

FIG. 2 is a schematic drawing of elements of a user device.

FIG. 3 illustrates an embodiment of the lighting system for videorecording and display featuring single-user foreground lightingelements.

FIG. 4 illustrates an embodiment of the lighting system for videorecording and display featuring an illumination indicator and lightingshown in a picture-in-picture display.

FIG. 5 illustrates an embodiment of the lighting system for videorecording and display featuring a lighting effect segmented behind auser's head.

FIG. 6 illustrates an embodiment of the lighting system for videorecording and display featuring foreground lighting on both userdevices.

FIG. 7 illustrates an embodiment of the lighting system for videorecording and display featuring a segmented background.

FIGS. 8-16 illustrate various lighting examples for use in the lightingsystem for video recording and display.

FIG. 17 illustrates an example of the lighting system used when taking aphotograph or recording a video using a front facing camera.

FIG. 18 illustrates an example of lighting options presented to a userwhen taking a photograph or recording a video using a front facingcamera.

FIG. 19 illustrates an example of using lighting elements to replace asegmented background when taking a photograph or recording a video usinga front facing camera.

FIG. 20 illustrates an example of a picture-in-picture display thatshows the effects of the lighting system when taking a photograph orrecording a video using a front facing camera.

DETAILED DESCRIPTION OF THE DRAWINGS

The present subject matter provides a computer-implemented lightingsystem for video recording and display 100 (the lighting system 100). Inthe example of the lighting system 100 shown in FIG. 1 , the lightingsystem 100 is implemented in a video conference call between a firstuser 101 using a first user device 103 and a second user 105 using asecond user device 107. Although the first user device 103 and thesecond user device 107 are shown as smartphones, it is understood thatthe first user device 103 and the second user device 107 may be any of asmartphone, a tablet, a video call specific portable device, a personalcomputer, a laptop, an office-based teleconferencing system, etc., aswill be understood by those skilled in the art based on the teachingsprovided herein.

FIG. 2 illustrates the various systems of the first user device 103shown in FIG. 1 . It is contemplated that the second user device 107 mayalso be represented by the elements shown in FIG. 2 .

FIG. 2 is a schematic diagram illustrating an example of the first userdevice 103 that may be used in the system shown in FIG. 1 . In theexample shown in FIG. 2 , the lighting system 100 runs as a videoconferencing application embodied in video conferencing software 164 onthe first user device 103. As shown in FIG. 2 , the first user device103 maybe a mobile device, such as a smartphone, running videoconferencing software 164 to provide the functionality described herein.A user may install the video conferencing software 164 on the first userdevice 103 via Apple's App Store, the Android Market, etc. The firstuser device 103 may include a wireless communication subsystem 120 tocommunicate with the second user device 107.

The first user device 103 may include a memory interface 102,controllers 109, such as one or more data processors, image processorsand/or central processors, and a peripherals interface 106. The memoryinterface 102, the one or more controllers 109 and/or the peripheralsinterface 106 can be separate components or can be integrated in one ormore integrated circuits. The various components in the first userdevice 103 can be coupled by one or more communication buses or signallines, as will be recognized by those skilled in the art.

Sensors, devices, and additional subsystems can be coupled to theperipherals interface 106 to facilitate various functionalities. Forexample, a motion sensor 108, a light sensor 110, and positioningsensors 112 can be coupled to the peripherals interface 106 tofacilitate orientation, lighting, and positioning functions. Othersensors 114 can also be connected to the peripherals interface 106, suchas a proximity sensor, a temperature sensor, a biometric sensor, orother sensing device, to facilitate related functionalities.

A camera subsystem 116 includes a physical camera (e.g., a chargedcoupled device (CCD) or a complementary metal-oxide semiconductor (CMOS)optical sensor) which can be utilized to facilitate camera functions,such as recording photographs and video clips. Modern smartphones andother mobile devices typically feature more than one physical cameraoperated by the camera subsystem 116. Such cameras may be located on thefront of the first user device 103—the side of the device with a touchscreen display 134 (e.g., front facing cameras 118) or rear of the firstuser device 103—the side opposite the touch screen display 134 (e.g.,rear facing cameras 119).

Communication functions can be facilitated through a network interface,such as one or more wireless communication subsystems 120, which caninclude radio frequency receivers and transmitters and/or optical (e.g.,infrared) receivers and transmitters. The specific design andimplementation of the communication subsystem 120 can depend on thecommunication network(s) over which the first user device 103 isintended to operate. For example, the first user device 103 can includecommunication subsystems 120 designed to operate over a GSM network, aGPRS network, an EDGE network, a 2G network, a 3G network, a 4G network,a 5G network, a Wi-Fi or Imax network, and a Bluetooth network. Inparticular, the wireless communication subsystems 120 may includehosting protocols such that the user device 20 may be configured as abase station for other wireless devices.

An audio subsystem 122 can be coupled to a speaker 124 and a microphone126 to facilitate voice-enabled functions, such as voice recognition,voice replication, digital recording, and telephony functions.

The I/O subsystem 128 may include a touch screen controller 130 and/orother input controller(s) 132. The touch screen controller 130 can becoupled to a touch screen display 134. The touch screen display 134 andtouch screen controller 130 can, for example, detect contact andmovement, or break thereof, using any of a plurality of touchsensitivity technologies, including but not limited to capacitive,resistive, infrared, and surface acoustic wave technologies, as well asother proximity sensor arrays or other elements for determining one ormore points of contact with the touch screen display 134. The otherinput controller(s) 132 can be coupled to other input/control devices136, such as one or more buttons, rocker switches, thumb-wheel, infraredport, USB port, and/or a pointer device such as a stylus. The one ormore buttons (not shown) can include an up/down button for volumecontrol of the speaker 124 and/or the microphone 126.

The memory interface 102 may be coupled to memory 104. The memory 104can include high-speed random access memory and/or non-volatile memory,such as one or more magnetic disk storage devices, one or more opticalstorage devices, and/or flash memory (e.g., NAND, NOR). The memory 104may store operating system instructions 140, such as Darwin, RTXC,LINUX, UNIX, OS X, iOS, ANDROID, BLACKBERRY OS, BLACKBERRY 10, WINDOWS,or an embedded operating system such as VxWorks. The operating systeminstructions 140 may include instructions for handling basic systemservices and for performing hardware dependent tasks. In someimplementations, the operating system instructions 140 can be a kernel(e.g., UNIX kernel).

The memory 104 may also store communication instructions 142 tofacilitate communicating with one or more additional devices, one ormore computers and/or one or more servers. The memory 104 may includegraphical user interface instructions 144 to facilitate graphic userinterface processing; sensor processing instructions 146 to facilitatesensor-related processing and functions; phone instructions 148 tofacilitate phone-related processes and functions; electronic messaginginstructions 150 to facilitate electronic-messaging related processesand functions; web browsing instructions 152 to facilitate webbrowsing-related processes and functions; media processing instructions154 to facilitate media processing-related processes and functions;GPS/Navigation instructions 156 to facilitate GPS and navigation-relatedprocesses and instructions; camera instructions 158 to facilitatecamera-related processes and functions; and/or other softwareinstructions 160 to facilitate other processes and functions (e.g.,access control management functions, etc.). The memory 104 may alsostore other software instructions controlling other processes andfunctions of the user device 20 as will be recognized by those skilledin the art. In some implementations, the media processing instructions154 are divided into audio processing instructions and video processinginstructions to facilitate audio processing-related processes andfunctions and video processing-related processes and functions,respectively. An activation record and International Mobile EquipmentIdentity (IMEI) 162 or similar hardware identifier can also be stored inmemory 104. As described above, the video conferencing software 164 isalso stored in the memory 104 and run by the controllers 109.

Each of the above identified instructions and applications cancorrespond to a set of instructions for performing one or more functionsdescribed herein. These instructions need not be implemented as separatesoftware programs, procedures, or modules. The memory 104 can includeadditional instructions or fewer instructions. Furthermore, variousfunctions of the first user device 103 may be implemented in hardwareand/or in software, including in one or more signal processing and/orapplication specific integrated circuits. Accordingly, the first userdevice 103, as shown in FIG. 2 , may be adapted to perform anycombination of the functionality described herein.

Aspects of the systems and methods described herein are controlled byone or more controllers 109. The one or more controllers 109 may beadapted run a variety of application programs, access and store data,including accessing and storing data in associated databases, and enableone or more interactions via the first user device 103. Typically, theone or more controllers 109 are implemented by one or more programmabledata processing devices. The hardware elements, operating systems, andprogramming languages of such devices are conventional in nature, and itis presumed that those skilled in the art are adequately familiartherewith.

For example, the one or more controllers 109 may be a central controlprocessing system utilizing a central processing unit (CPU), memory 104and an interconnect bus. The CPU may contain a single microprocessor, orit may contain a plurality of microcontrollers 109 for configuring theCPU as a multi-processor system. The memory 104 include a main memory,such as a dynamic random access memory (DRAM) and cache, as well as aread only memory, such as a PROM, EPROM, FLASH-EPROM, or the like. Thesystem may also include any form of volatile or non-volatile memory 104.In operation, the main memory is non-transitory and stores at leastportions of instructions for execution by the CPU and data forprocessing in accord with the executed instructions.

The one or more controllers 109 may further include appropriateinput/output ports for interconnection with one or more output displays(e.g., monitors, printers, touch screen displays 134, motion-sensinginput device 108, etc.) and one or more input mechanisms (e.g.,keyboard, mouse, voice, touch, bioelectric devices, magnetic reader,RFID reader, barcode reader, touchscreen 134, motion-sensing inputdevice 108, etc.) serving as one or more user interfaces for theprocessor. For example, the one or more controllers 109 may include agraphics subsystem to drive the output display (e.g., touch screendisplay 134). The links of the peripherals to the system may be wiredconnections or use wireless communications.

Although summarized above as a smartphone-type implementation, thoseskilled in the art will recognize that the one or more controllers 109also encompasses systems such as other mobile devices, host computers,servers, workstations, network terminals, PCs, and the like. In fact,the use of the term controller is intended to represent a broad categoryof components that are well known in the art.

A first example illustrating an application of the lighting system 100shown in FIG. 3 includes a first user 101 and a second user 105participating in a video conference with each other using theirsmartphones 103 and 107. In this example, each user views a live videofeed of the other user on a front facing display 166 (e.g., a touchscreen display 134) while simultaneously capturing a live video feed ofthemselves using the front facing camera 118.

The lighting system 100 shown in FIG. 3 includes a first set of lightingelements 168 displayed as foreground lighting elements 168. As shown inFIG. 3 , the first user device 103 includes a front facing camera 118and a front facing display 166. When running the video conferencingsoftware 164, the front facing display 166 provides a main displayelement 170 and a picture-in-picture display element 172. The maindisplay element 170 displays a first live video feed featuring thesecond user 105 captured by the second user device 107 and thepicture-in-picture display element 172 displays a second live video feedfeaturing the first user 101 captured by the first user device 103.

Similarly, the second user device 107 includes a front facing camera 118and a front facing display 166. When running the video conferencingsoftware 164, the front facing display 166 provides a main displayelement 170 and a picture-in-picture display element 172. The maindisplay element 170 displays the second live video feed featuring thefirst user 101 captured by the first user device 103 and thepicture-in-picture display element 172 displays the first live videofeed featuring the second user 105 captured by the second user device107.

In the example shown in FIG. 3 , the first set of lighting elements 168are displayed as foreground lighting elements 168 lining the perimeterof the front facing display 166. The lighting elements 168 shown arebright, white pixels that illuminate the face of the first user 101 toimprove the lighting of the second live video being captured by thefirst user device 103. The lighting elements 168 in this example areshown as foreground lighting elements 168, meaning they are displayed infront of the videos shown on the main display element 170 and thepicture-in-picture display element 172.

The example shown in FIG. 4 is similar to the example shown in FIG. 3 ,but adds an illumination indicator 174 and also displays the first setof lighting elements 168 in the picture-in-picture display element 172on the second user device 107. Accordingly, the second user 105 is ableto see that the first user device 103 is using a first set of lightingelements 168 to illuminate the first user 101 and can additionally seewhat the second user 105 looks like on the first user device 103 incombination with the first set of lighting elements 168.

The example shown in FIG. 5 is similar to the example shown in FIG. 4 ,but the first set of lighting elements 168 are shown as backgroundlighting elements 168 on the first user device 103. In other words, thelighting elements 168 disappear behind the video of the second user 105.The lighting elements 168 can be displayed as background lightingelements 168 when one of the first user device 103 and the second userdevice 107 segment the second user 105 from the background 176 in thefirst live video feed.

The example shown in FIG. 6 is similar to the example shown in FIG. 1 ,but the foreground lighting elements 168 are displayed on both of thefirst user device 103 and the second user device 107.

In the example shown in FIG. 7 , the second user 105 is segmented fromthe background 176 of the first video and the entirety of the background176 is replaced with background lighting elements 168 on the second userdevice 107.

FIGS. 8-16 illustrate various examples of lighting elements 168 invarious configurations and geometries. The varied examples shown in thedrawings illustrate how the lighting elements 168 may be foregroundlighting elements 168, may be background lighting elements 168, mayreplace segmented backgrounds 176, may be provided on one or both userdevices simultaneously, etc.

Turning now to FIG. 17 , an example is provided in which a first user101 uses a first user device 103 including a front facing camera 118 anda front facing display 166 to take a selfie photograph or record aselfie-style video. Just as shown in the example provided with referenceto FIG. 3 , the first set of lighting elements 168 are displayed asforeground lighting elements 168 lining the perimeter of the frontfacing display 166. The lighting elements 168 shown are bright, whitepixels that illuminate the face of the first user 101 to improve thelighting of the content to be captured by the first user device 103.

FIG. 18 illustrates an example of lighting activation options 178presented to the user 101 when taking a photograph or recording a videousing a front facing camera 118. In the example shown in FIG. 18 , whenthe first user device 103 identifies that the ambient lightingconditions are poor for capturing a well-lit photograph or video, thefront facing display 166 presents a notification 180 to the user 101stating “Poor Lighting Conditions Detected.” The display 166 furtherpresents three lighting element activation options 178 for activatinglighting, including “yes,” “auto,” and “no.” In this example, theoptions are provided with respect to background lighting elements 168,though the concepts described herein apply equally to foreground andother lighting elements 168. When the user 101 selects the “yes” option,the lighting elements 168 are activated regardless of the ambientlighting conditions. When the user 101 selects the “auto” option, thelighting elements 168 are activated when the lighting system 100determines that the lighting quality of the content to be captured willbe improved by activating the lighting elements 168. When the user 101selects the “no” option, the lighting elements 168 are not activatedregardless of the ambient lighting conditions.

FIG. 19 illustrates an example of using lighting elements 168 to replacea segmented background 176 when taking a photograph or recording a videousing a front facing camera 118. As shown in the example provided inFIG. 19 , the display 166 presents a notification 180 that the a“Background Lighting Activated” mode is active. In this mode, asegmented background 176 in the main display element 170 is replaced bylighting elements 168. As further shown in FIG. 19 , the display 166provides a picture-in-picture display element 172 that provides the user101 a representation of the content to be captured. As shown, therepresentation in the picture-in-picture display element 172 does notinclude the lighting elements 168, but instead shows the background thatwill be captured by the front facing camera 118. In this embodiment, theuser 101 is able to see the content that is to be captured while stillbenefiting from the use of the lighting elements 168.

FIG. 20 illustrates an example of a picture-in-picture display element172 that shows the effects of the lighting system 100 when taking aphotograph or recording a video using a front facing camera 118. Asshown, the picture-in-picture display element 172 presents the user witha representation of the content that would be captured under thelighting conditions in the absence of the activation of the lightingelements 168, which in this example are shown as segmented backgroundlighting elements 168. This presentation enables the user 101 todetermine whether the lighting effects 168 are necessary to provide theappropriate lighting conditions. As further shown in FIG. 20 , thedisplay 166 presents the user 101 with an illumination indicator 174 toindicate that the lighting elements 168 are active.

It should be noted that various changes and modifications to thepresently preferred embodiments described herein will be apparent tothose skilled in the art. Such changes and modifications may be madewithout departing from the spirit and scope of the present invention andwithout diminishing its attendant advantages.

I claim:
 1. A computer-implemented lighting system for video recording and display comprising: a first user device featuring a front-facing camera, a front-facing display, a processor, a memory, and a networking interface; and a second user device featuring a front-facing camera, a front-facing display, a processor, a memory, and a networking interface; wherein the first user device's processor: receives a first video captured by the second user device's front-facing camera; displays the first video on the first user device's display, wherein a portion of the first video is replaced by one or more lighting elements on the first user device's display; and captures a second video using the first user device's front-facing camera, wherein the second video is illuminated during capture, in part, by the one or more lighting elements on the first user device's display.
 2. The system of claim 1, wherein the first video includes video of an object of interest and a background and one of the first user device's processor and the second user device's processor: identifies and segments the object of interest from the background in the first video; and replaces at least a portion of the segmented background in the first video with the one or more lighting elements.
 3. The system of claim 1, wherein the operation of the one or more lighting elements on the first user device's display is user controlled.
 4. The system of claim 1, wherein the operation of the one or more lighting elements on the first user device's display is automatically controlled by the first user device's processor.
 5. The system of claim 1, wherein the second user device's processor: receives a second video captured by the first user device's front-facing camera; displays the second video on the second user device's display, wherein a portion of the second video is replaced by a second set of one or more lighting elements on the second user device's display; and captures a first video using the second user device's front-facing camera, wherein the first video is illuminated during capture, in part, by the second set of one or more lighting elements on the second user device's display.
 6. The system of claim 1, wherein the lighting elements overlap a picture-in-picture element shown on the first user device's display.
 7. The system of claim 1, wherein the lighting elements displayed on the first user device's display are replicated on a picture-in-picture display on the second user device's display such that the picture-in-picture display on the second user device's display replicates the first user device's display.
 8. The system of claim 1, wherein the lighting elements displayed on the first user device's display are not replicated on a picture-in-picture display on the second user device's display such that the picture-in-picture display on the second user device's display shows the first video without any representation of the lighting elements displayed on the first user device's display.
 9. The system of claim 1, wherein a brightness of the one or more lighting elements on the first user device's display automatically adjust based on ambient lighting conditions detected by the first user device.
 10. The system of claim 1, wherein a brightness of the one or more lighting elements on the first user device's display and a brightness of the first video displayed on the first user device's display are independently adjustable.
 11. The system of claim 1, wherein a brightness of the second user device's is controlled by the first user device's processor.
 12. The system of claim 1, wherein the one or more lighting elements on the first user device's display replace a segmented background element in the first video.
 13. The system of claim 1, wherein the one or more lighting elements on the first user device's display are displayed behind a segmented head and shoulders captured by the second user device's front-facing camera.
 14. A computer-implemented lighting system for video recording and display comprising: a first user device featuring a front-facing camera, a front-facing display, a processor, a memory, and a networking interface; and a second user device featuring a front-facing camera, a front-facing display, a processor, a memory, and a networking interface; wherein the first user device's processor: receives a first video captured by the second user device's front-facing camera; displays the first video on the first user device's display, wherein a portion of the first video is replaced by a first set of one or more lighting elements on the first user device's display; automatically adjusts a brightness of the first set of one or more lighting elements on the first user device's display based on ambient lighting conditions detected by the first user device; and captures a second video using the first user device's front-facing camera, wherein the second video is illuminated during capture, in part, by the one or more lighting elements on the first user device's display; wherein the second user device's processor: receives a second video captured by the first user device's front-facing camera; displays the second video on the second user device's display, wherein a portion of the second video is replaced by a second set of one or more lighting elements on the second user device's display; automatically adjusts a brightness of the second set of one or more lighting elements on the second user device's display based on ambient lighting conditions detected by the second user device; and captures a first video using the second user device's front-facing camera, wherein the first video is illuminated during capture, in part, by the one or more lighting elements on the second user device's display.
 15. The system of claim 14, wherein, the first video includes video of an object of interest and a background and one of the first user device's processor and the second user device's processor: identifies and segments the object of interest from the background in the first video; and replaces at least a portion of the segmented background in the first video with the first set of one or more lighting elements; and the second video includes video of an object of interest and a background and one of the first user device's processor and the second user device's processor: identifies and segments the object of interest from the background in the second video; and replaces at least a portion of the segmented background in the second video with the second set of one or more lighting elements.
 16. The system of claim 14, wherein a brightness of the first set of one or more lighting elements on the first user device's display and a brightness of the first video displayed on the first user device's display are independently adjustable and a brightness of the second set of one or more lighting elements on the second user device's display and a brightness of the second video displayed on the second user device's display are independently adjustable.
 17. The system of claim 14, wherein a brightness of the first set of one or more lighting elements is additionally controlled by the second user device's processor and wherein a brightness of the second set of one or more lighting elements is additionally controlled by the first user device's processor.
 18. The system of claim 14, wherein a brightness of the first set of one or more lighting elements is additionally controlled by the first user device's processor and wherein a brightness of the second set of one or more lighting elements is additionally controlled by the second user device's processor.
 19. The system of claim 14, wherein the first set of one or more lighting elements on the first user device's display are displayed behind a segmented head and shoulders captured by the second user device's front-facing camera and the second set of one or more lighting elements on the second user device's display are displayed behind a segmented head and shoulders captured by the first user device's front-facing camera.
 20. The system of claim 14, wherein the first set of one or more lighting elements displayed on the first user device's display are replicated on a picture-in-picture display on the second user device's display such that the picture-in-picture display on the second user device's display replicates the first user device's display and second set of one or more lighting elements displayed on the second user device's display are replicated on a picture-in-picture display on the first user device's display such that the picture-in-picture display on the first user device's display replicates the second user device's display. 